Intro to Sentiment Analysis

Reference book: https://www.tidytextmining.com/

Sentiment analysis can be thought of as the exercise of taking a sentence, paragraph, document, or any piece of natural language, and determining whether that text’s emotional tone is positive, negative or neutral. One way to analyze the sentiment of a text is to consider the text as a combination of its individual words and the sentiment content of the whole text as the sum of the sentiment content of the individual words.

When human readers approach a text, we use our understanding of the emotional intent of words to infer whether a section of text is positive or negative, or perhaps characterized by some other more nuanced emotion like surprise or disgust.

library(tidytext)
library(tidyverse)

Registered S3 methods overwritten by 'dbplyr':
  method         from
  print.tbl_lazy     
  print.tbl_sql      
── Attaching packages ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.1 ──
✓ ggplot2 3.3.5     ✓ purrr   0.3.4
✓ tibble  3.1.5     ✓ dplyr   1.0.7
✓ tidyr   1.1.4     ✓ stringr 1.4.0
✓ readr   2.0.2     ✓ forcats 0.5.1
── Conflicts ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()

library(textdata)

The tidytext package contains sentiments dataset. The three general-purpose lexicons are:

• AFINN from Finn Arup Nielsen,

• bing from Bing Liu and collaborators, and

• nrc from Saif Mohammad and Peter Turney.

The above different sentiment lexicons, based on unigrams (i.e. single words) are derived from a single English word and are assigned different scores of positive/negative sentiments. Each lexicon scores a different way from the others.

• AFINN scores a word with a number, which may range from -5 to +5.

• bing scores a word as either positive or negative.

• nrc categorizes a word under sentiment type categories such as positive, negative, anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.

All of this information is tabulated in the sentiments dataset, and tidytext provides a function get_sentiments() to get specific sentiment lexicons without the columns that are not used in that lexicon.

# sample of sentiments
sample_n( get_sentiments("afinn") , 5)

# sample of sentiments
sample_n( get_sentiments("nrc") , 5)

# sample of sentiments
sample_n( get_sentiments("bing"), 5)

There are also some domain-specific sentiment lexicons available, constructed to be used with text from a specific content area.

Not every English word is in the lexicons because many English words are pretty neutral. It is important to keep in mind that these methods do not take into account qualifiers before a word, such as in “no good” or “not true”; a lexicon-based method like this is based on unigrams only.

One last caveat is that the size of the chunk of text that we use to add up unigram sentiment scores can have an effect on an analysis. A text the size of many paragraphs can often have positive and negative sentiment averaged out to about zero, while sentence-sized or paragraph-sized text often works better.

Songs Lyrics Example

Read data

# data from https://raw.githubusercontent.com/reisanar/datasets/master/BB_top100_2015.csv
lyrics_2015 <- read_csv("BB_top100_2015.csv")

Rows: 100 Columns: 6
── Column specification ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (3): Song, Artist, Lyrics
dbl (3): Rank, Year, Source

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

lyrics_2015 <- select(lyrics_2015, -Year, -Source)

Take a look at the data you loaded:

head(lyrics_2015)

Let us focus on artists that have more than 1 song in the Top 100

# print artist that appear more than once in the list
lyrics_2015[duplicated(lyrics_2015$Artist),]

Goal: study the sentiment of the top 10 songs lyrics

lyrics_2015 %>% 
  filter(Rank %in% 1:10)

First, let us convert the text to the tidy format using unnest_tokens(). Notice that we choose the name word for the output column from unnest_tokens(). This is a convenient choice because the sentiment lexicons and stop word datasets have columns named word; performing inner joins and anti-joins is thus easier.

lyrics_2015 %>% 
  filter(Rank %in% 1:10) %>%
  unnest_tokens(word, Lyrics)

Notice that we did not remove any stop-words!

lyrics_2015 %>%
  filter(Rank %in% 1:10) %>%
  unnest_tokens(word, Lyrics, token = "words") %>% filter(!word %in% stop_words$word, str_detect(word, "[a-z]"))

We could explore this dataset by first, checking the frequency of words in the top 10 songs lyrics.

First let us store our tidy text data set in a new object called bb_top_10

# BB top 10 songs lyics in 2015 (no stop-words)
bb_top_10 <- lyrics_2015 %>% 
  filter(Rank %in% 1:10) %>% unnest_tokens(word, Lyrics, token = "words") %>% 
  filter(!word %in% stop_words$word, str_detect(word, "[a-z]"))

Check the most used words:

bb_top_10 %>%
  group_by(word) %>%
  summarise(uses = n()) %>%
  arrange(desc(uses)) %>%
  head(10)

With the information above, we can graphically show a sample of frequent words

 bb_top_10 %>%
  group_by(word) %>%
  summarise(uses = n()) %>%
  arrange(desc(uses)) %>%
  slice(1:15) %>%
  ggplot() +
  geom_bar(aes(x = word, y = uses), stat = "identity") +
  coord_flip() +
  theme_minimal()

Sentiment Analysis

Much as removing stop words is an anti-join operation, performing sentiment analysis is an inner join operation.

bb_top_10 %>%
  inner_join(get_sentiments("bing")) %>%
  count(Song, sentiment) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentiment = positive - negative)

Joining, by = "word"

And a visualization of it can tell us about the “sentiment” of each song as a whole:

bb_top_10 %>%
  inner_join(get_sentiments("bing")) %>%
  count(Song, sentiment) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentiment = positive - negative) %>%
  ggplot() +
  geom_bar(aes(x = reorder(Song, sentiment), y = sentiment), stat = "identity") +
  coord_flip() +
  labs(x = "", title = "Sentiment Analysis of Songs", subtitle = "Top 10 Billboard songs in 2015") +
  theme_minimal()

Joining, by = "word"

What if we want to find out how the sentiment of the song varies as the song progresses? Let us use now the afinn lexicon

bb_top_10 %>%
  inner_join(get_sentiments("afinn"))

Do you want to download:
 Name: AFINN-111 
 URL: http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010 
 License: Open Database License (ODbL) v1.0 
 Size: 78 KB (cleaned 59 KB) 
 Download mechanism: https 

1: Yes
2: No

1

trying URL 'http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/6010/zip/imm6010.zip'
Content type 'application/zip' length 16227 bytes (15 KB)
==================================================
downloaded 15 KB

Joining, by = "word"

# add an index column to keep track of the words used 
bb_top_10 %>%
  group_by(Song) %>%
  mutate(index = row_number() ) %>%
  inner_join(get_sentiments("afinn"))

Joining, by = "word"

Notice that several songs in this top 10 list, contain many words that are not part of the lexicon, and therefore were not scored:

bb_top_10 %>%
  group_by(Song) %>%
  mutate(index = row_number() ) %>%
  inner_join(get_sentiments("afinn")) %>%
  ggplot() +
  geom_col(aes(x = index, y = value)) +
  facet_wrap(~Song, scales = "free", ncol = 2) +
  labs(title = "Sentiment Analysis of Songs using AFINN lexicon",
       subtitle = "Top 10 Billboard songs in 2015",
       x = "") +
  theme_minimal()

Joining, by = "word"

Focus on one artist

Goal: perform sentiment analysis using the nrc lexicon for songs by Taylor Swift

lyrics_2015 %>%
  filter(Artist == "taylor swift")

First, let us convert the text to the tidy format using unnest_tokens(). Notice that we choose the name word for the output column from unnest_tokens(). This is a convenient choice because the sentiment lexicons and stop word datasets have columns named word; performing inner joins and anti-joins is thus easier.

lyrics_2015 %>%
  filter(Artist == "taylor swift") %>%
  unnest_tokens(word, Lyrics)

Notice that we did not remove any stop-words!

We could explore this dataset by first, checking the frequency of words in the top 10 songs lyrics.

First let us store our tidy text data set in a new object

# taylor swift songs (no stop-words)
swift <- lyrics_2015 %>%
  filter(Artist == "taylor swift") %>%
  unnest_tokens(word, Lyrics) %>%
  anti_join(stop_words)

Joining, by = "word"

Check the most used words:

swift %>%
  group_by(word) %>%
  summarise(uses = n()) %>%
  arrange(desc(uses))

With the information above, we can graphically show a sample of frequent words

swift %>%
  group_by(word) %>%
  summarise(uses = n()) %>%
  arrange(desc(uses)) %>%
  slice(1:15) %>%
  ggplot() +
  geom_bar(
    aes(x = reorder(word, uses), y = uses), stat = "identity") +
  coord_flip() +
  labs(y = "frequency of use", x = "") +
  theme_minimal()

Sentiment Analysis

Using the nrc lexicon

swift %>%
  inner_join(get_sentiments("nrc")) %>%
  group_by(sentiment) %>%
  count() %>%
  ggplot() +
  geom_bar(
    aes(x = reorder(sentiment, n), y = n), stat = "identity") +
  labs(title = "Sentiment analysis using nrc lexicon", subtitle = "Lyrics of Songs by Taylor Swift on 2015 Billboard Top 100",  x = "", y = "word-count") +
  theme_minimal()

Do you want to download:
 Name: NRC Word-Emotion Association Lexicon 
 URL: http://saifmohammad.com/WebPages/lexicons.html 
 License: License required for commercial use. Please contact Saif M. Mohammad (saif.mohammad@nrc-cnrc.gc.ca). 
 Size: 22.8 MB (cleaned 424 KB) 
 Download mechanism: http 
 Citation info:

This dataset was published in Saif M. Mohammad and Peter Turney. (2013), ``Crowdsourcing a Word-Emotion Association Lexicon.'' Computational Intelligence, 29(3): 436-465.

article{mohammad13,
author = {Mohammad, Saif M. and Turney, Peter D.},
title = {Crowdsourcing a Word-Emotion Association Lexicon},
journal = {Computational Intelligence},
volume = {29},
number = {3},
pages = {436-465},
doi = {10.1111/j.1467-8640.2012.00460.x},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8640.2012.00460.x},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8640.2012.00460.x},
year = {2013}
}
If you use this lexicon, then please cite it. 

1: Yes
2: No

1

trying URL 'http://saifmohammad.com/WebDocs/NRC-Emotion-Lexicon.zip'
Content type 'application/zip' length 24436570 bytes (23.3 MB)
==================================================
downloaded 23.3 MB

Joining, by = "word"

Show words associated to some of the sentiments:

swift_words <- swift %>%
  inner_join(get_sentiments("nrc")) %>%
  group_by(sentiment, word) %>%
  count(mycount = n()) %>%
  distinct() %>% filter(sentiment %in% c("negative", "anger",
                                         "positive", "trust"))

Joining, by = "word"

# plot
ggplot(data = swift_words, aes(label = word)) +
  ggrepel::geom_label_repel(
    aes(
      x = word,
      y = rnorm(nrow(swift_words)),
      label = word),
    direction = "both",
    box.padding = 0.04,
    segment.color = "transparent",
    size = 3) +
  facet_wrap( ~ sentiment, ncol = 2) +
  labs(x = "", y = "") +
  theme(
    axis.text.y = element_blank(),
    axis.text.x = element_blank(),
    axis.ticks = element_blank(),
    panel.grid = element_blank(),
    panel.background = element_blank())